Automating Identification of Code Snippets for Library Suggestion Models

ABSTRACT

A method, system, and apparatus are disclosed for adding library models to a library knowledge base by defining a template for a library configuration file that conveys information about each library model, custom inputs and code snippets to facilitate library comparison operations, and education content for the library model, where the library configuration file template may be automatically filled by populating selected data fields in the template with information identifying the library model and a set of custom inputs for the library model, and then passing a set of test cases from the set of custom inputs to an input/output matching engine to evaluate a repository of code snippets and identify a set of functionally similar code snippets for inclusion one or more data fields in the template.

CROSS REFERENCE TO RELATED APPLICATIONS

U.S. patent application Ser. No. ______, entitled “Library ModelAddition” by Tushar Makkar, Attorney Docket No. T00260DF, filed on sameday herewith, which is incorporated by reference in its entirety as iffully set forth herein.

U.S. patent application Ser. No. ______, entitled “Automating Generationof Library Suggestion Engine Models” by Tushar Makkar, Attorney DocketNo. T00274DF, filed on same day herewith, which is incorporated byreference in its entirety as if fully set forth herein.

U.S. patent application Ser. No. ______, entitled “AutomatingIdentification of Test Cases for Library Suggestion Models” by TusharMakkar, Attorney Docket No. T00276DF, filed on same day herewith, whichis incorporated by reference in its entirety as if fully set forthherein.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention is directed in general to field of informationprocessing. In one aspect, the present invention relates generally to asoftware program development tool, method, and apparatus in a dataprocessing system.

Description of the Related Art

Computer programmers, developers, coders and/or software engineerswrite, test, debug, and maintain computer software or code instructions,called computer programs, which computers must follow to perform theirfunctions. When writing or making changes to computer program to addressnew or unique technical challenges, programmers often create new,original and unique programming source code which can, unfortunately,suffer from performance limitations and other sub-optimalities. Forexample, a programmer's unique source code may include a number ofundetected software bugs or otherwise suffer from low quality orrobustness if the code has not been thoroughly tested, or may havereduced performance functionality if the code is outdated or has notbeen updated or maintained. Unique source code is often also bloated andless readable than desired due to inefficiencies in the computerprogram's use of memory, disk space, processing power, or other hardwareresources whilst making only dubious user-perceptible improvements orsuffering from feature creep. Programmers can address many of theseproblems by using libraries of basic code that can be modified orcustomized for a specific application, thereby yielding more reliableand consistent programs and increasing the programmer's productivity.However, there are significant difficulties with using libraries in thata significant amount of effort, knowledge, and experience is needed inorder to correctly identify a suitable library from among thousands ofstored library files, to understand the performance and codingrequirements of the library, and to make any required code adjustmentsthereto. In addition, the responsibility for using a library typicallyresides with the programmer or code reviewer, making it difficult totrack and enforce library usage. There are similar challenges withdeveloping, updating and expanding existing library databases sincethere are no standardized mechanisms for generating, vetting, and addinglibrary functions to an existing library knowledge base. In the absenceof control procedures for reviewing and validating proposed libraryadditions, an uploaded library file can corrupt or damage the libraryknowledge base. Thus, while the use of libraries is considered a bestpractice for software development, the existing solutions for addinglibrary functions to promote library use are extremely difficult at apractical level by virtue of the difficulty in identifying, uploading,adding, adopting, and modifying libraries.

SUMMARY OF THE INVENTION

A system, apparatus, and methodology are described for efficientlyimproving code reuse and improving codebase maintainability byautomating the addition of library functions to a library recommendationengine which identifies library functions for replacement orsubstitution of source code which is written by developers. In selectedembodiments, a human-readable data serialization language, such as YAML,is used to specify a library configuration file format for onboardinglibrary functions, where the library configuration file represents thedata needed to recognize a single function in the library. In an exampleYAML configuration file, the library function is specified by thedeveloper/library owner in terms of a library information (e.g., libraryname, function name, library descriptors, documentation link(s) for thelibrary function, library code link(s), method signature, transformfunction snippets, etc.), sample inputs and/or outputs for the libraryfunction (e.g., inputs for blackbox and whitebox matching engines),functionally similar code snippets to help establish a pattern for thelibrary code, and educational content for the library function. Thoughthe specified content of the library configuration file may be input bythe developer, the configuration file may be automatically generated orauto-templated, such as by scraping the webpage of a given librarydocumentation link to populate the configuration file with specifiedvalues (e.g., library function signature, documentation link, etc.) in aspecified format (e.g., YAML) without developer intervention at all. Inaddition or in the alternative, the library configuration input file maybe automatically populated with functionally similar code snippets foruse with a whitebox matching engine by using library function sampletest cases as inputs to a blackbox matching engine to analyze opensource projects and thereby identify the functionally similar codesnippets. Once the library configuration file format is specified foronboarding, the configuration file is submitted to a library modeladdition engine which validates the input library configuration file toensure that the file is in the correct format and satisfies theconstraints provided by the library recommendation engine. In selectedembodiments, a YAML validator may be implemented as a python packagewhich validates whether the content of YAML configuration file iscorrect or not. For example, the code snippets and transform functionsnippets from the library configuration file may be compiled as part ofthe validation process to check the correctness of the code snippets.Once validated, the approved library function is uploaded to the libraryknowledge base where it may be used to generate customized codesuggestions for library function substitutions for a programmer'ssubmitted source code. As described more fully hereinbelow, the libraryfunction substitution recommendations are generated by pruning the inputsource code to identify candidate code snippets from the source codewhich are matched with recommended library functions for substitution inthe submitted source code. Selected embodiments of the disclosed system,apparatus, and methodology use machine learning, natural languageprocessing (NLP), and/or artificial intelligence (AI) in combinationwith static and/or dynamic code analysis techniques to automaticallyanalyze code and yield library substitution opportunities. As a resultof the match processing, the programmer may be presented with one ormore library function recommendations which may include code lines frominput source code files along with code lines from the library functionsuggestion, alone or in combination with additional library functioninformation identifying the code improvement recommendation and/or codereduction resulting from the library function recommendation and/oreducational tutorial information relating to the implementation of thelibrary function recommendation.

The objects, advantages and other novel features of the presentinvention will be apparent from the following detailed description whenread in conjunction with the appended claims and attached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention may be understood, and its numerous objects,features and advantages obtained, when the following detaileddescription of a preferred embodiment is considered in conjunction withthe following drawings.

FIG. 1 is a simplified block diagram of a data processing system foradding library models to a library knowledge base which is used togenerate library function recommendations in accordance with selectedembodiments of the present disclosure.

FIG. 2 illustrates a library model addition workflow for use ingenerating library reuse recommendations in accordance selectedembodiments of the present disclosure.

FIG. 3 illustrates a simplified flow chart showing the logic forsubmitting validated library models which may be recommended for libraryfunction substitutions to a developer in accordance selected embodimentsof the present disclosure.

FIG. 4A is a first example screen shot of a user interface of a librarymodel addition engine which illustrates the evaluation of input sourcecode files in accordance selected embodiments of the present disclosure.

FIG. 4B is a second screen shot of a user interface of a librarysuggestion engine which illustrates a code reduction opportunity for aninput source code file in accordance selected embodiments of the presentdisclosure.

FIG. 4C is a third screen shot of a user interface of a librarysuggestion engine which illustrates a library function recommendationfor an input source code file in accordance selected embodiments of thepresent disclosure.

FIG. 5 is a screen shot of user interface of a library administratorconsole which illustrates the automatically populated data fields of alibrary configuration file representing a library function in accordanceselected embodiments of the present disclosure.

FIG. 6 is a simplified block diagram of a general-purpose computer inaccordance with selected embodiments of the present disclosure.

DETAILED DESCRIPTION

A library model addition engine, methodology, and apparatus aredescribed for adding a candidate library model to a library knowledgebase by generating and validating a library configuration filecontaining data needed to recognize the library model, includingautomatically generated code snippets that are functionally similar tothe candidate library model and that are identified by passing libraryfunction test cases to an input/output matching engine which evaluatesopen source projects to identify functionally similar code snippets foruse with an internal or whitebox matching engine. While various detailsare set forth in the following description, it will be appreciated thatthe present invention may be practiced without these specific details.For example, selected aspects are shown in block diagram form, ratherthan in detail, in order to avoid obscuring the present invention. Someportions of the detailed descriptions provided herein are presented interms of algorithms or operations on data within a computer memory. Suchdescriptions and representations are used by those skilled in the dataprocessing arts to describe and convey the substance of their work toothers skilled in the art. In general, an algorithm refers to aself-consistent sequence of steps leading to a desired result, where a“step” refers to a manipulation of physical quantities which may, thoughneed not necessarily, take the form of electrical or magnetic signalscapable of being stored, transferred, combined, compared, and otherwisemanipulated. It is common usage to refer to these signals as bits,values, elements, symbols, characters, terms, numbers, or the like.These and similar terms may be associated with the appropriate physicalquantities and are merely convenient labels applied to these quantities.Unless specifically stated otherwise as apparent from the followingdiscussion, it is appreciated that throughout the description,discussions using terms such as processing, computing, calculating,determining, displaying or the like, refer to the action and processesof a computer system, or similar electronic computing device, thatmanipulates and/or transforms data represented as physical, electronicand/or magnetic quantities within the computer system's registers andmemories into other data similarly represented as physical quantitieswithin the computer system memories or registers or other suchinformation storage, transmission or display devices.

Referring now to FIG. 1, a simplified block diagram illustrates anexemplary data processing system 100 for adding library models (e.g.,Library Function 1) to a library knowledge base for use in generatinglibrary function recommendations with one or more server/computersystems 10 having processor(s) 11, memory 12, and associated databasestorage devices 21 which may be connected together over a suitableconnection link 34 or network, such as a private wide area network (WAN)or the Internet (not shown). To illustrate the operative functionalityof the data processing system 100, the server memory 12 may be used tostore the program module functionality for the library suggestion engine13 and library model addition engine 16. As described hereinbelow, thelibrary suggestion engine 13 and library model addition engine act uponthe input source code files 22 and/or data from the database storagedevices 21 to add library functions to the library knowledge base 28and/or to generate recommendations for replacing source code written bydevelopers with library functions stored in the knowledge base 28. Theoperation of the library suggestion engine 13 to transform input sourcecode files 22 into validated code snippets 25 for matching with libraryfunctions 26 is shown with processing steps 31-33, though it will beappreciated that the underlying data may be stored in the databasestorage device 21 and/or memory 12.

In selected illustrative embodiments, the server/computer system 10 mayinclude a library suggestion engine 13 that is embodied as a QuestionAnswering (QA) system to use NLP, machine learning, and/or artificialintelligence processing techniques which are augmented with themechanisms of the illustrative embodiments described hereafter. In a QAsystem implementation, the library suggestion engine 13 may be embodiedas a computer program stored in memory 12 which uses one or moreprocessors 11 to query a structured or unstructured knowledge orinformation database 21 which includes a library knowledge base 28.Input source code files 22 are available from an external system and/ormay be stored in memory 12 and/or in the database storage device 21.Illustrated as being stored in memory 12, the library suggestion engine13 may be configured to assess input source code files 22 against alibrary knowledge base 28 for purposes of suggestion library functionsas replacement for one or more lines of code from the input source codefiles. The configuring of the library suggestion engine 13 may includeproviding application specific hardware, firmware, or the like tofacilitate the performance of the operations and generation of theoutputs described herein with regard to the illustrative embodiments. Inaddition or in the alternative, the configuring of the librarysuggestion engine 13 may include storing software applications in one ormore storage devices and loaded into memory of a computing device, suchas server 10, for causing one or more hardware processors (e.g., 11) ofthe computing device to execute the software applications that configurethe processors to perform the operations and generate the outputsdescribed herein with regard to the illustrative embodiments. Moreover,any combination of application specific hardware, firmware, softwareapplications executed on hardware, or the like, may be used withoutdeparting from the spirit and scope of the illustrative embodiments.

The library suggestion engine 13 may evaluate the input source codefiles to reduce or prune the number of potential candidate source codesnippets for library suggestion by applying natural language processing(NLP) techniques 31 to reduce the processing burden for making libraryfunction recommendations. To this end, the library suggestion engine 13may be provided with a pruning engine 14 for analyzing the input sourcecode files 22 (e.g., Source Code File A, Source Code File B, Source CodeFile C, . . . Source Code File n) using any suitable technique toidentify candidate code snippets 24 (e.g., Source Code File A and SourceCode File B) and remove others which are not likely candidates forlibrary function substitution (e.g., Source Code File C and Source CodeFile n).

As a preliminary step in identifying the candidate code snippets 24, thepruning engine 14 may apply NLP techniques 31 to generate source codefeature vectors 23 by checking for the presence of predetermined wordsin the input source code files 22 and assigning a corresponding weight.For example, the pruning engine 14 may be configured to apply suitableNLP techniques 31 to make the source code feature vectors 23, includingbut not limited to a tokenization step (which breaks each source codefile into words, phrases, symbols and other elements known as tokens), alowercasing step (which normalizes the source code files by lowercasingthem), a stopwords removal step (which removes grammatical words, knownas stopwords, and java-specific words from the source code files), afull form conversion step (which converts short form words, such as stror inp, in the source code file to full form, such as string or input),a semantic sense addition step (which adds contextual or semanticinformation to key words in the source code file, such as adding “loop$”if the keyword is in loop or adding “condition$” if the keyword isinside a conditional statement), a stemming step (which reduces words inthe source code file to their root form by removing inflectional wordendings), a splitting step (which converts combined words into separatewords), and/or a similar sense detection step (which uses Wordnet to addsimilar words, such as synonyms, to the words in the source code file).In selected embodiments, the pruning engine 14 may also be configured touse an NLP or machine learning process which applies a topical model,such as a Latent Dirichlet Allocation (LDA) module or Latent SemanticAnalysis (LSA) module, to extract topics in the input source code files22. In addition, the pruning engine 14 may be configured to use a termfrequency-inverse document frequency (TF-IDF) module to identifyimportant keywords in the input source code files 22. Once the importantkeywords or topics from the input source code files are identified withthe LDA, LSA, and/or TF-IDF algorithms, the pruning engine 14 may beconfigured to combine the results into net result source code featurevectors 23, such as by using a dot product of the priority of eachkeyword with their priority value found from LSI, LDA, and TF-IDFalgorithms, respectively.

Once the source code feature vector files 23 are generated, the pruningengine 14 may be further configured to identify candidate code snippets24 by applying pruning process 32 to identify candidate code snippets 24from the input source code files 22 that are likely candidates forlibrary function substitution opportunities. For example, the pruningengine 14 may be configured with a heuristic engine or module whichcomputes a (one time) pruning threshold (e.g., Pruning Threshold 1) foreach library function (e.g., Library Function 1) from the libraryfunction code snippets (e.g., LF Code Snippet(s) 1) which perform thesame or essentially the same function as the library function. Inselected embodiments, the pruning threshold may be calculated as theminimum of the dot product of vector achieved from the library functioncode snippets from permutation of the library knowledge base 28. Thispruning threshold and the average vector may then be used to categorizewhether a source code file should be further analyzed as a candidatecode snippet 24.

To provide additional details for an improved understanding of selectedembodiments of the present disclosure for pruning input source codefiles, the following use case scenario is provided for analyzing twoinput source code files 22, namely Source Code File A=a.java and SourceCode File B=b.java. In this example, the first input source code file,a.java, is set forth below:

a.java package example; class a{   public static void main(String args[]){     int b;     String a = “hello”;     b(a);    System.out.println(a);   } }

In addition, the second input source code file, b.java, is set forthbelow:

b.java: package example; import java.io.*; importjava.io.BufferedReader; class b {   public static String readFile(Stringfile) throws IOException {     /* This function is used for reading afile to string     */     BufferedReader br;     br = newBufferedReader(new FileReader(file));     StringBuilder sb = newStringBuilder( );     while(br.ready( )){       sb.append(br.readLine());       sb.append(‘\n’);     }     br.close( );     returnsb.toString( );   } }

Continuing with this example use case scenario, the NLP processing 31 ofthe input source files 22 performed by the pruning engine 14 would beused to generate a feature vector consisting of a string of identifiedkeywords or topics, each having an associated or computed weightingfactor. For example, a source code feature vector 23 (e.g., FeatureVector File B) for the input source code file 22 (e.g., Source Code FileB) would be generated from the sample source code b.java as thefollowing vector structure:

[‘loop$readi’, 0.3247174394233218], [‘tostr’, 0.3903609231317347],[‘loop$append’, 0.4470159023826545] [‘loop$readlin’,0.4609584097830426], [‘builder’, 0.5936101105253638], [‘readlin’,0.6050065274587427], [‘file’, 0.6998090230209926], [‘stringbuild’,0.7123321326304367], [‘append’, 0.7481612762292236], [‘loop$line’,0.8873805425301969], [‘read’, 0.9581944056457393], [‘bufferedread’,1.659810014435204], [‘line’, 1.6774363021832421], [‘buffer’,1.705919490241575]

Using the source code feature vectors 23, each corresponding source codefile 22 is evaluated against the different library functions (e.g.,Library Functions 1-i) stored in the library knowledge base 28. To thisend, each library function (e.g., Library Function 2) stores basicinformation about library function, an associated library function codesnippet (e.g., LF Code Snippet 2), a pruning threshold value (e.g.,Pruning Threshold 2), a pruning vector (e.g., Pruning Vector 2)specifying a feature vector for the library function, along withadditional matching parameters described more fully hereinbelow, such asdistance and size threshold values and an Ngram threshold value.

As disclosed herein, the basic library function information for LibraryFunction 2 may include the signature for the library function, a linkwhere the library code is hosted (e.g., github link), the documentationlink, educational content, etc. In addition, an example LF code snippet2 which perform same work as Library Function 2 for apache commonsIOUtils to string function can be:

public static String lib_readFile12(InputStreamReader is) throwsIOException {  StringBuilder s = new StringBuilder( );  BufferedReaderbr = new BufferedReader(is);  while (br.ready( )) {  s.append(br.readLine( )+“\n”);  }  String s2 = s.toString( ); br.close( );  return s2; }

In addition, an example pruning threshold value (e.g., Pruning Threshold2) and pruning vector (e.g., Pruning Vector 2) for Library Function 2may be stored in the library knowledge base 28 with the following vectorstructure:

‘apache.commons.io.IOUtils.toString’: [0.26090225563909775,[[[‘ngramsi’, 0.0392156862745098], [‘array’, 0.05084745762711865],[‘add’, 0.05084745762711865], [‘outstream’, 0.05172413793103448],[‘method$loop$reader’, 0.05298245614035088], [‘method$loop$readlin’,0.05298245614035088], [‘method$loop$read’, 0.05298245614035088],[‘file_nam’, 0.057167356450603515], [‘sock’, 0.0603448275862069],[‘data’, 0.06707317073170732], [‘arraylist’, 0.06779661016949153],[‘loop$file’, 0.06936247840182302], [‘hasnextlin’, 0.07407407407407407],[‘loop$nextlin’, 0.07407407407407407], [‘loop$hasnextlin’,0.07407407407407407], [‘nextlin’, 0.07407407407407407], [‘loop$reader’,0.09126873586033678], [‘method$loop$append’, 0.10596491228070176],[‘fileinputstream’, 0.11433471290120703], [‘list’, 0.11864406779661017],[‘fileread’, 0.15390307066222714], [‘method$loop$line’,0.15894736842105261], [‘inputstreamread’, 0.2571415633386791],‘scanner’, 0.25925925925925924], [‘input’, 0.31993278485297943],[‘readi’, 0.3247174394233218], [‘loop$readi’, 0.3247174394233218],[‘stream’, 0.3544155434736691], [‘tostr’, 0.3903609231317347],[‘loop$append’, 0.4470159023826545], [‘stringbuff’, 0.4587506367647674],[‘loop$readlin’, 0.4609584097830426], [‘loop$read’, 0.5219340195391402],[‘builder’, 0.5936101105253638], [‘readlin’, 0.6050065274587427],[‘file’, 0.6998090230209926], [‘stringbuild’, 0.7123321326304367],[‘append’, 0.7481612762292236], [‘loop$line’, 0.8873805425301969],[‘read’, 0.9581944056457393], [‘bufferedread’, 1.659810014435204],[‘line’, 1.6774363021832421], [‘buffer’, 1.705919490241575], [‘reader’,2.164677513480203]]]]

In this example vector structure, the first term is the pruningthreshold (e.g., Pruning Threshold 2=0.26090225563909775), and theremaining vector structure includes a sequence of feature vectors wherethe first term is a key word or topic from the library function codesnippet, and the second term is the strength or weighting factorassigned to the first term. Stated more generally, the pruning vectorfor a given library function may be represented as: {pruning_threshold,[[keyword_to_search, strength_of_the_keyword_in_given_context]]}.

Similar to the NLP vector processing 31 described above, a uniquepruning vector (e.g., Pruning Vector 1-i) for each library function(e.g., Library Functions 1-i) may be generated by applying NLPtechniques to identify predetermined words from the correspondinglibrary function/LF code snippet and assign a corresponding weight.Example feature vector algorithms include tokenization steps,lowercasing steps, stopword removal steps, full form conversion steps,semantic sense addition steps, stemming steps, splitting steps, similarsense detection steps, LDA processing steps, LSA processing steps,and/or TF-IDF processing steps.

Continuing with this example use case scenario, the pruning engine 14may be configured to reduce or filter the input source code files 22down to a smaller set of candidate code snippets 24 with a pruningprocess 32 that evaluates the source code feature vectors 23 againstlibrary function information in the library knowledge base 28. In thepruning process 32, each source code feature vector 23 may be comparedfor similarity to the pruning vectors in each library function using asimilarity threshold value. For example, the input source code featurevector (e.g., Feature Vector File B) generated from an sample inputsource code file (e.g., Source Code File B=b.java) may be compared witheach of the library function pruning vectors (e.g., Pruning Vectors 1-i)to determine if the sample input source code file (e.g., Source CodeFile B=b.java) qualifies as a candidate code snippet 24. With thesefeature vector examples of this use case scenario, the pruning process32 is applied by the pruning engine 14 to identify Source Code FileB=b.java as a candidate code snippet by evaluating Feature Vector File Bagainst the pruning vector (e.g., Pruning Vector 2) for the LibraryFunction 2=“apache.commons.io.IOUtils.toString” stored in the libraryknowledge base 28. This evaluation may be performed as a dot productcomputation of common terms from Feature Vector File B and PruningVector 2 by multiplying the weights for the common terms and summing theresults to determine if the resulting value exceeds the pruningthreshold (e.g., Pruning Threshold 2=0.26090225563909775). In this case,the dot production computation value is 13.1299337163802074136 andexceeds the value for the Pruning Threshold 2, indicating that thecorresponding input source code file (e.g., Source Code File B)qualifies as a candidate code snippet 24. However, if the dot productcomputation from Feature Vector File C and the library function pruningvectors does not exceed the any of the pruning threshold values for theLibrary Functions 1-i, then the corresponding input source code file(e.g., Source Code File c) does not qualify as a candidate code snippet24.

Once the candidate code snippets 24 are identified, the librarysuggestion engine 13 may read and analyze the candidate code snippets 24by applying NLP matching techniques 33 to extract features from thecandidate code snippets 24 for comparison matching with featuresextracted from a given library function. To this end, the librarysuggestion engine 13 may be provided with a matching engine 15 foridentifying validated code snippets 25 from the input source code whichmatch with library functions in the library knowledge base 28. Inselected embodiments, the matching engine 15 may employ a white boxmatching (WBM) module to perform fuzzy or internal match processing 33which reads and analyzes the candidate code snippets 24 to extractpredetermined features for matching with the features extracted from agiven library function, thereby generating validated code snippets 25(e.g., from Source Code File B) which can be replaced by a matchinglibrary function 26 (e.g., from Library Function 2). In addition or inthe alternative, the matching engine 15 may employ a black box matching(BBM) module to perform input/output matching which injects sharedinputs to candidate code snippets 24 and library function code snippetsto detect matching outputs, thereby generating validated code snippets25 (e.g., from Source Code File B) which can be replaced by a matchinglibrary function 26 (e.g., from Library Function 2).

Once the matching library functions 26 are identified, the librarysuggestion engine 13 may present library function recommendations 27 tothe program developer with suggestions for swapping the validated codesnippets 25 with the matching library functions 26. In selectedembodiments, a library function recommendation 27 may include thevalidated source code snippets from the input source code files (e.g.,Source Code File B) along with a visual indication that suggests alibrary function (e.g., Library Function 2) for replacement orsubstitution. For example, a first user interface display screen mayshow an input source code file (e.g., Source Code File B) with thevalidated code snippet 25 highlighted or otherwise visually set off fromthe remaining lines of code in the input source code file, such as byincluding a user interaction link which opens a second user interfacedisplay screen to show information relating to the matching libraryfunction 26 (e.g., Library Function 2).

In selected illustrative embodiments, the library model addition engine16 may also be embodied in the server/computer system 10 as a QA systemto use NLP, machine learning, and/or artificial intelligence processingtechniques which are augmented with the mechanisms of the illustrativeembodiments described herein. In a QA system implementation, the librarymodel addition engine 16 may be embodied as a computer program stored inmemory 12 which uses one or more processors 11 to query a structured orunstructured knowledge or information database 21 which includes alibrary knowledge base 28. The configuring of the library model additionengine 16 may include providing application specific hardware, firmware,or the like to facilitate the performance of the operations andgeneration of the outputs described herein with regard to theillustrative embodiments. In addition or in the alternative, theconfiguring of the library model addition engine 16 may include storingsoftware applications in one or more storage devices and loaded intomemory of a computing device, such as server 10, for causing one or morehardware processors (e.g., 11) of the computing device to execute thesoftware applications that configure the processors to perform theoperations and generate the outputs described herein with regard to theillustrative embodiments. Moreover, any combination of applicationspecific hardware, firmware, software applications executed on hardware,or the like, may be used without departing from the spirit and scope ofthe illustrative embodiments.

However configured, the library model addition engine 16 is connected toreceive program inputs, including information describing each libraryfunction or model 1, functionally similar code snippets 2, and educationcontent 3, are available from an external system and/or may be stored inmemory 12 and/or in the database storage device 21. Illustrated as beingstored in memory 12, the library model addition engine 16 may beconfigured onboard a library model with a user interface input screen toenter the library information 1, code snippets 2, and education content3 as a library configuration file 4 using a YAML-based file format. Eachlibrary configuration file 4 represents the data needed to recognize asingle function in the library knowledge base 28, and should not includemultiple function configurations. In selected embodiments, the libraryconfiguration file 4 may be automatically generated into ahuman-readable format which is parsable by software (e.g., YAML) byusing an extractor 17 to extract, scrape, or otherwise construct thelibrary configuration file 4 by populating the following sections:

-   -   Name of the library (e.g., library_name),    -   Name of the library function (e.g., library_function_name),    -   Library Descriptors which mention the property of a particular        library function,    -   Online link to the documentation page for the library function        (e.g., documentation_link), logo image link (e.g.,        logo_image_link), library code link (e.g., library_code_link),        and/or maven link (e.g., maven_link),    -   Method signature (e.g., signature),    -   One or more custom inputs (e.g., custom_inputs) that can be        passed to the library functions,    -   Transform functions for converting complex type to simple type,    -   A list of conditions (e.g., equal_function_True) that will        determine a valid match when the expression evaluates to True,    -   A list of conditions (e.g., equal_function_False) that will        determine a valid match when the expression evaluates to False,        and    -   One or more functionally similar code snippets (e.g.,        code_snippets) that will help establish a pattern for the code.

In accordance with selected embodiments of the present disclosure, theextractor 17 may include program code and/or modules 5-7 for use inautomatically generating functionally similar code snippets 8. Asdepicted, the extractor 17 is coupled to retrieve sample test cases 5extracted from the custom inputs section of the library configurationfile. The test cases 5 may be provided as inputs to a blackbox matchingengine 6 which performs input/output matching where the test cases 5 areanalyzed against a repository of open source code bases 7 by injectingthe test cases 5 into the codes of each open source project 7. As aresult, the functionally similar code snippets 8 can be identified asthe sample code snippets 2 in the library configuration file 4 for thecandidate library model. With this approach, the developer is relievedfrom the burden of searching for code bases and identifying promisingcode snippets that can be used for internal or fuzzy matching evaluationof candidate library models.

The library model addition engine 16 may also be configured with avalidator 18 for evaluating a submitted input library configuration file4 to ensure that the file is in the correct format and satisfies theconstraints provided by the library recommendation engine. In selectedembodiments, a YAML validator 18 may be implemented as a python packagewhich validates whether the content of YAML configuration file iscorrect or not. One part of the validator package may include a YAMLformat validator to check the YAML format and fields for correctness andvalidity, to confirm that the identified links are working links, tocheck that a minimum number of code snippets are included. Another partof the validator package may be a fuzzy/internal matching enginevalidator which checks the validity of code snippets in terms ofcompilability of each code snippet. Yet another part of the validatorpackage may be an input/output matching engine validator which checksthe validity of transform functions and custom inputs against the methodsignature. If the transform function is not compilable or not inaccordance with the method signature, the validator 18 identifies thelibrary configuration file as faulty. In addition, the validator 18 maycheck if the custom inputs provided are correct (i.e., they satisfy themethod signature). In addition, the validator 18 may check that a givenJAR for a library function will generate outputs with respect to thegiven set of custom inputs. Using the validator 18, developers are ableto smoothly generate and test the library models locally on theirsystem. This has made their life a lot easier than before and thisprovides luxury for checking the library models locally on their system.

As will be appreciated, once the server/computer system 10 is configuredto implement the library suggestion engine 13 and/or library modeladdition engine 16, the server/computer system 10 becomes a specializedcomputing device specifically configured to implement the mechanisms ofthe illustrative embodiments and is not a general purpose computingdevice. Moreover, as described hereafter, the implementation of themechanisms of the illustrative embodiments improves the functionality ofthe computing device and provides a useful and concrete result thatfacilitates the addition of library functions to a library knowledgebase 28 and/or the generation of library function recommendations for aprogram developer by pruning input source code 22 into candidate codesnippets 24 which are then matched as validated code snippets 25 withcorresponding library functions 26 based on a measure of fuzzy and/orinput/output matching similarity for presentation to the programdeveloper as library function recommendations 27.

To provide a contextual understanding for selected embodiments of thepresent disclosure, reference is now made to FIG. 2 which illustrates aworkflow 200 for generating library reuse recommendations 240 from inputsource code and binaries 208 in accordance selected embodiments of thepresent disclosure. In selected embodiments, the depicted workflow 200may be implemented in whole or in part with a data processing system(such as shown in FIG. 1) and/or a single integrated circuit (e.g., asystem-on-chip) or a plurality of integrated circuits to implement oneor more processors with local memory and database storage that areinterconnected and interoperate using any suitable techniques withoutdeparting from the present invention.

However implemented, the workflow 200 receives input code (block 208)which may be checked into the system as source code and binary filescreated by the program developer. An additional input to the work flow200 is the library function information 204 provided by the libraryknowledge base 202 which stores library function information, codesnippets which perform the same work as a library function, pruningthreshold and pruning vector information, distance and size thresholdinformation, and Ngram threshold information. To manage and curate thelibrary function information 204 in the library knowledge base 202, thelibrary model developer 230 may submit configuration files 232 whichrepresent the data needed to recognize each library function in thelibrary. For example a configuration file for a single library functionmay include the name of the library (e.g., library_name), the name ofthe library function (e.g., library_function_name), library descriptorsinformation (e.g., information specifying the property of a particularlibrary function), an http url link to the documentation (e.g.,documentation_link), a method signature (e.g., signature), one or morecustom inputs that can be passed to the library functions, one or moretransform functions for converting complex type to simple type, a listof conditions that will determine a valid match when the expressionevaluates to True (e.g., equal_function_True), a list of conditions thatwill determine a valid match when the expression evaluates to False(e.g., equal_function_False), and or more code snippets that will helpestablish a pattern for the code (e.g., code_snippets). In this way, thelibrary knowledge base 202 has only the best of library functions, andcan also support customized library code for customer needs so that theycan import their particular library usage within their organization.

Upon check-in to the workflow 200, the input code is pruned or parsed bythe pruning engine 210 to identify candidate code snippets for librarysuggestion opportunities. While any suitable pruning approach may beused, selected embodiments may employ a tokenization engine 212 whichuses NLP processing to break the input code into tokens 213 (e.g.,tokenized source code) which are passed to the heuristic engine 214. Theheuristic engine 214 is also connected to receive library functioninformation 204 from the library knowledge base 202 which stores datathat may be curated and standardized according to frequency in terms ofusage, number of issues, commits, contributions, and various otherfactors. In selected embodiments, the heuristic engine 214 pools thetokens into candidate code snippets 215 using various techniques, suchas polygraph, latency, and indexing, LDA, etc. The resulting candidatecode snippets 215 are then passed to a matching engine 220 which is inturn composed of two different engines—a fuzzy or internal matchingengine 222 and an input/output matching engine 228.

Fuzzy/Internal or White Box Matching

At the fuzzy/internal matching engine 222 (also referred to as a “whitebox matching” engine), each received candidate code snippet 215 isprocessed for matching with library feature vectors from the libraryfunction information 206 based on their structural and execution flowsimilarity. To this end, the fuzzy/internal matching engine 222 may beprovided with an Abstract Syntax Tree (AST) matching algorithm 224 foridentifying validated code snippets from the input source code whichmatch with library functions in the library knowledge base 202. Inselected embodiments, the AST matching algorithm 224 may employ a syntaxtree to represent the abstract syntactic structure of the candidate codesnippets 215 and library function code snippets 206 from the libraryknowledge base 202 that are written in a source code programminglanguage. Each node of the tree denotes a construct occurring in thesource code. In addition, the syntax is “abstract” in not representingevery detail appearing in the real syntax. For instance, groupingparentheses are implicit in the tree structure, and a syntacticconstruct like an if-condition-then expression may be denoted by meansof a single node with three branches. In selected embodiments, the ASTmatching algorithm 224 may be embodied with the Clone Digger duplicatecode detection tool which performs anti-unification analysis of the ASTof the library function code snippets 206 and the candidate codesnippets 215 being validated. In operation, the AST matching algorithm224 constructs an abstract syntax tree (AST) for each code snippet beinganalyzed in any desired source code language, such as Python, Java, andother programming languages, such as Java 8. For example, the ASTmatching algorithm 224 may be implemented by creating a wrapper on topof Clone Digger to extend the AST matching for Java 8 code snippets withimproved accuracy and efficiency.

For improved accuracy for of the AST matching algorithm 224, the ASTsize threshold and distance threshold values may be adjusted or tweakedfor each different library function, and then stored while onboardingany new library function in the library knowledge base 202 for automatedusage. As will be appreciated, the AST size threshold value specifiesthe minimum size of the nodes in each abstract syntax tree, while theAST distance threshold values specifies the maximum distance which isallowed for two ASTs to qualify as matching. In accordance with thepresent disclosure, the AST distance threshold value for each libraryfunction may be calculated from the library function code snippets inthe library knowledge base 202 which are permuted and applied to theClone Digger anti-unification algorithm. In addition, the AST sizethreshold value may be calculated as the minimum number of nodes in thelibrary knowledge base 202 multiplied by a constant factor which is lessthan 1. The Clone Digger tool used to implement the AST matchingalgorithm 224 may also be modified to allow changes to the variable nameand function name parameters and/or to remove the report generationfeature.

Continuing with an example use case scenario to illustrate how the fuzzyor internal matching engine 222 compares features from a candidate codesnippet 215 and library function code snippet 206, the AST matchingalgorithm 224 is used to construct an AST for each candidate codesnippet 215 and library function code snippet 206 and then apply aclustering algorithm to find similar code snippets, such as by using theant-unification algorithm from the Clone Digger implementation of theAST matching algorithm 224. In addition, Ngram matching algorithm 226 isused to break the candidate code snippet (e.g., Source Code FileB=b.java) into class and then into system level instructions as setforth below:

indexOf:(Ljava/lang/String;)IShowWeekdayR.main:([Ljava/lang/String;)V,15 0indexOf:(Ljava/lang/String;)IShowWeekdayR.main:([Ljava/lang/String;)V,38 0indexOf:(Ljava/lang/String;)IShowWeekdayR.main:([Ljava/lang/String;)V,61 0split:(Ljava/lang/String;)[Ljava/lang/String;  ShowWeekdayR.main:([Ljava/lang/String;)V,72 0java/lang/Integer.parseInt:(Ljava/lang/String;)I  ShowWeekdayR.main:([Ljava/lang/String;)V,80 0java/lang/Integer.parseInt:(Ljava/lang/String;)I  ShowWeekdayR.main:([Ljava/lang/String;)V,86 0

After converting both the library function code snippets and thecandidate code snippets being analyzed, the fuzzy/internal matchingengine 222 finds the similar code snippets at function level.

To provide additional match processing, the fuzzy/internal matchingengine 222 may also be provided with an Ngram matching algorithm 226 foridentifying validated code snippets from the input source code whichmatch with library functions in the library knowledge base 202. Inselected embodiments, the Ngram matching algorithm 226 may employ acontiguous sequence of n items formed from the ASM files which in turnare extracted from Class files to represent the internal structure ofthe candidate code snippets 215 and library function code snippets 206from the library knowledge base 202 that are written in a source codeprogramming language. In operation, the Ngram matching algorithm 226extracts n-grams from a given candidate code snippet 215 for comparisonwith the Ngrams extracted from library function code snippets 206 fromthe library knowledge base 202. In selected embodiments, the Ngrammatching algorithm 226 may be embodied with the Agec execution-semanticclone detection tool which analyzes the library function code snippets206 and the candidate code snippets 215 being validated. In operation,the Ngram matching algorithm 226 applies an abstract interpretation tobytecode as a static analysis in order to generate n-grams from thecandidate code snippets 215, detects the same n-grams from distinctplaces of the library function code snippets 206, and then reports thesen-grams as code clones.

For improved accuracy for of the Ngram matching algorithm 226, the Ngramthreshold value may be adjusted or tweaked for each different libraryfunction, and then stored while onboarding any new library function inthe library knowledge base 202 for automated usage. As will beappreciated, the Ngram threshold value specifies the minimum number ofNgrams which need to be matched in order for the candidate code snippetto be validated. In accordance with the present disclosure, the Ngramthreshold value for each library function may be calculated by findingthe minimum of maximum number of Ngrams present in the library knowledgebase 202. In addition, the Agec clone detection embodiment of the Ngrammatching algorithm 226 may be modified to provide ASM-level operatorsand/or to add literals and parent function features for similar codedetection. In other embodiments, the Agec clone detection tool used toimplement the Ngram matching algorithm 226 may also be extended to workwith input data types (such as array, Integer, String, etc.), to allowfor multiple function invocations within the same file, and/or to autogenerate custom inputs by extracting test cases from the libraryfunction test cases.

In selected embodiments, the AST matching algorithm 224 and Ngrammatching algorithm 226 may be further modified for improved accuracy byadding a literal or constant Ngram to require a strict matchingalgorithm in cases where the literal constants play an important rolefor a code snippet to work similar to a library function. For example,when a removeExtension instruction has a literal constant of ‘.’ whichwhen replaced with something like ‘I’ in the candidate code snippet beanalyzed, then the matching algorithm should not yield a librarysubstitution opportunity as removeExtension, even though most of the ASTand Ngrams are similar with the library knowledge base 202.

Input/Output or Black Box Matching

In addition to performing fuzzy/internal matching, the input/outputmatching engine 228 (also referred to as a “black box matching” engine)is configured to inject shared inputs into the candidate code snippets215 and the library functions via JARs which are extracted via mavenlinks presented in the library function information 206 to see if thecandidate code snippet output is similar to the library function codesnippet output. Based on the output similarity, the input/outputmatching engine 228 assigns a similarity rating. If the outputs match,the candidate code snippet 215 being analyzed is validated for possiblereplacement with the matching library function.

Continuing with an example use case scenario to illustrate how theinput/output matching engine 228 injects a shared input into a candidatecode snippet 215 and library function code snippet 206 to compare theoutput results, consider the example of the following input candidatecode snippet 215 being compared at the input/output matching engine 228is set forth below:

--code-- if (path.indexOf(“\\”) == −1) {    answer = path;   } answer =path1; --code-- At the input/output matching engine 228, this inputcandidate code snippet is converted to: public java.lang.StringtestMethod(String testMethodInput_0, String testMethodInput_1) throwsException {String path = testMethodInput_0;   if (path.indexOf(“\\”) ==−1) {    return path;    }   String returnTestMethodVar =testMethodInput_1;    return returnTestMethodVar; }

The input/output matching engine 228 may then use an interface program,such as Py4J, to pass inputs into this function, and the results arematched with corresponding library function results which are calculatedby running jar with same parameters.

The main challenge for performing input/output matching is to extractthe right code snippets for matching analysis. The performance ofinput/output matching at the function level of granularity rarely workssince most of the library suggestion opportunities are hidden withincode segments that are difficult to recognize. To address thislimitation, the input/output matching engine 228 may be configured tocheck each and every code snippet line by treating them as start and endlines. While doing this, the code snippet AST is analyzed to maintain asymbol table. Based on the input parameters to the input/output matchingengine 228 and the return type of the library function being analyzedfor suggestion, validated code snippets 229 may be extracted. To thisend, the input/output matching engine 228 extracts all the basic importsused in the candidate code snippet 215 being analyzed and the candidatecode snippet is templatized by forming appropriate class with functiondefinition. Next, the candidate code snippet is checked to see if it iscompilable. If the candidate code snippet is compilable, a predeterminedset of inputs is injected into the candidate code snippet and theresulting output is checked to see if they are similar or identical tothe outputs from the library function.

By combining the outputs from the fuzzy/internal matching engine 222 andinput/output matching engine 228, the matching engine 220 identifiesvalidated code snippets with matching library functions 229 which areoutput as library reuse recommendations 240.

Presenting Library Function Recommendations

The results of the match processing by the matching engine 220 arepresented to the programmer as one or more library functionrecommendations which include may include code lines from input sourcecode files (e.g., validated code snippets) along with code lines fromthe recommended library function. To assist the developer with therecommended library substitution, the developer may also be providedwith additional library function information identifying the codeimprovement recommendation and/or code reduction resulting from thelibrary function recommendation and/or educational tutorial informationrelating to the implementation of the library function recommendation.

Adding Library Functions to the Library Knowledge Base

To support the addition of new library functions to the accumulatedlibrary knowledge base, the workflow 200 is configure to receive andvalidate a library configuration file 232 created by the programdeveloper 230 when adding a library configuration model 231 to thelibrary knowledge base 202. Generally speaking, the libraryconfiguration file 232 includes library function information 233,functionally similar code snippets 234 which perform the same work asthe library function, sample inputs and outputs for the library function235, and educational content 236. For example a library configurationfile 232 may be formatted as a YAML format file to describe a singlecandidate library function in terms of a library name (e.g.,library_name), library function name (e.g., library_function_name), oneor more library descriptors describing properties of the libraryfunction, a documentation link for the library function (e.g.,documentation_link), a method signature (e.g., signature), one or morecustom inputs that can be passed to the library functions, one or moretransform functions for converting complex type to simple type, a listof conditions that will determine a valid match when the expressionevaluates to True (e.g., equal_function_True), a list of conditions thatwill determine a valid match when the expression evaluates to False(e.g., equal_function_False), and or more code snippets that will helpestablish a pattern for the code (e.g., code_snippets). In this way, thelibrary knowledge base 202 has only the best of library functions, andcan also support customized library code for customer needs so that theycan import their particular library usage within their organization.

Configuration File Structure

Library Name and Library Function Name

To provide additional details for an improved understanding of selectedembodiments of the library configuration file 232, the followingconfiguration file structure is provided for representing the dataneeded to recognize a library function. In this example, the “libraryname” data structure field (e.g., library_name: “org.apache.commons.io”)provides a simple label that identifies the library that the functionbelongs to. The best practice here is to use the root package name ofthe library. In addition, the “Library Function Name” data structurefield (e.g., library_function_name:“org.apache.commons.io.IOUtils.toString”) provides a simple label toidentify the function in the library that is being configured.

Library Descriptors

In addition, the disclosed configuration file structure may include a“library descriptors” data structure with fields identifying one or moreoptional descriptors which mention the property of a given library andare used for showing results in a proper format. In an exampleembodiment, the “library descriptors” data structure may include a “Typeof Library Function” library descriptor (e.g., “code_type”) whichmentions the type of code which a given library function can replace.The “library descriptors” data structure may include an “Embedding VideoTutorial Link” library descriptor (e.g., “embed_video_link”) whichdefines a HTTP url which is a video tutorial explaining how to replacecode using this library function. In addition, a “Maven Repository Link”library descriptor (e.g., “maven_link”) may be included which defines anHTTP url link for the mavenlink to the library. In similar fashion, a“Library Code Link” library descriptor (e.g., library_code_link”) maydefine a HTTP url which is the link to the source code of the libraryhosted on revision control system. If desired, the “library descriptors”data structure may also include a library's “Logo Image Link” librarydescriptor (e.g., “logo_image_link”) which defines an HTTP url which isthe link to uploaded/available logo of library. The “librarydescriptors” data structure should also include a “Summary Of LibraryFunction” library descriptor (e.g., “summary”) which provides a one linesummary to what the library function does, and a “Number Of Lines”library descriptor (e.g., “number_of_lines”) which lists the number oflines of code which would be added when replaced with the function.Generally, the “Number of Lines” descriptor is set to “1” unless thereis a need to initialize parameters. In addition, the “librarydescriptors” data structure may include a “Description” librarydescriptor (e.g., “description”) which is a dictionary which provides adetailed description about the library by specifying the advantages andcode snippet details. For example, the dictionary may have a “Code” key(e.g., “code”) which contains the code snippet which shows usage of thelibrary function. In addition, the dictionary may have a “Advantage” key(e.g., “advantage”) which is an array with multiple parameters, namely“name,” “rating,” and “description.” As their respective names signify,“name” represents the name of the advantage, “rating” (an integralvalue) may represent a rating value (e.g., on a 5-star rating system),and “description” may represent the reason behind giving the rating.

To provide additional details for an improved understanding of selectedembodiments of the present disclosure, the following use case scenariois provided as an example of the library descriptors for the followinglibrary function:

, public static String concat(String basePath,   StringfullFilenameToAdd)where the “library descriptors” data structure is:

code_type: “Filepath Manipulation” embed_video_link:“https://www.youtube.com/embed/7_dogg9zK2k” library_code_link:“https://github.com/apache/commons-io” maven_link:“http://mvnrepository.com/artifact/commons-io/commons- io/2.4”logo_image_link: “http://s20.postimg.org/4g0cz3ayh/apache.png” summary:“The code snippet takes input as file paths and try to join them usingseparators.” number_of_lines: 1 description:  code:“FilenameUtils.concat(base_path, path_to_join)”  advantages:  - name:“Maintainability”  rating: 3  description: “Code is less complex,clearer and easy to understand since Apache Commons FilenameUtils is apopular library.Hence the code becomes more maintainable”  - name:“Robustness”  rating: 4  description: “Apache Foundation is a veryactive community and hence the chances of bugs in the library code isless and the fixes are frequent making the code robust”  - name:“Documentation”  rating: 3  description: “There is complete and ampledocumentation given for Apache Commons FilenameUtils Library”  - name:“Test Coverage”  rating: 3  description: “Apache Commons FilenameUtilshas a comprehensive set of developer tests, providing assurance of thelibrary quality”  - name: “Maturity”  rating: 3  description: “Thelibrary has been under active development since 2002. It is a mature andreliable code base”

Documentation Link

In addition, the disclosed configuration file structure may include a“Documentation Link” data structure (e.g., documentation_link:“https://commons.apache.org/proper/commons-io/apidocs/org/apache/commons/io/IOUtils.html#toString(java.io.Reader)”)with fields specifying the online url link for the documentation page ofthe library function (not the full library).

Method Signature

The disclosed configuration file structure may also include a “methodsignature” data structure which specifies the signature for the libraryfunction. In selected embodiments, the “method signature” data structuremay include a “method parameter types” part (which defines the type forthe method input parameter) and a “method return type” part (whichdefines the return type of the method call), both of which are Javaobjects. All the types should be having their fully-qualified names. Toprovide an improved understanding of selected embodiments of the “methodsignature” data structure, reference is now made to a first examplesingle parameter method signature:

public static String toString(Reader input)           throws IOExceptionwhere the corresponding configuration for the method signature would be:

signature:   method_parameter_types:     - java.io.Reader   // Type ofthe input parameter   return_type:     - java.lang.String   // Returntype of the method

For an additional example “method signature” data structure, referenceis now made to a signature definition for a two-input method, such asStringUtils.join, having a two-parameter method signature:

public static String join(String[ ] array,     char separator)in which case the corresponding configuration for the method signaturewould be: signature:

signature:   method_parameter_types:     - java.lang.String [ ]     -char   return_type:     - java.lang.String

For another example “method signature” data structure, reference is nowmade to a signature definition for a three-input method, such asFilenameUtils.wildCardMatch, having a three-parameter method signature:

public static boolean wildcardMatch(String fileName, StringwildCardMatcher, IOCase caseSensitivity)in which case the corresponding configuration for the method signaturewould be:

method_parameter_types:     - java.lang.String     - java.lang.String    - org.apache.commons.io.IOCase   return_type:     - boolean

Custom Inputs

As disclosed herein, the configuration file structure may also include a“custom inputs” data structure which is used to provide custom inputs tothe library function. In an example embodiment, the “custom inputs” datastructure may include a “custom inputs” field which defines custominputs that can be passed to the library function. The number of inputsin each instance of a custom input should be equal to the number ofmethod_parameter_types in the signature section. In addition, the typeof each input value in a custom input instance should match thecorresponding type in method_parameter_types. The “custom inputs” datastructure may include an “input” field which specifies each instance ofa custom input.

For simple types of library functions (such as String, int, bool ortheir array counterparts), the “custom inputs” data structure can bewritten in a straightforward manner. For example, a use case scenario ofcustom inputs for the library function, StringUtils.join, would be thecode given below.

custom_inputs: - input: # StringUtils.join takes in two parameters -[“a”, “b”, “c”] # First one is an array of Strings - “_” # Second one isa character variable - input: # Number and type of parameters each inputhas should be equal to what the library function has - [“d”, “e”, “f”] - “.”

In selected embodiments, the library configuration model 231 may beconfigured to support a predetermined set simple types of libraryfunctions, such as java.lang.String, java.lang.CharSequence,java.lang.String [ ], java.lang.Character, java.lang.Integer,java.lang.Integer [ ], java.lang.Boolean, int, int [ ], char, andBoolean. However, for other types of library functions apart thesupported simple types, the program developer may be required to writecustom inputs. For example, if the input for a library function is acomplex object, or an object from a class present in an external JAR, orinput types, such as InputStream, BufferedReader etc., then there is nostraightforward way to express them.

In order to be able to send complex data types to custom inputs, thelibrary configuration model 231 may be configured with transformfunctions that build complex types from the basic types. In such cases,the transform function may be implemented as a Java function snippetthat takes in a basic type and then uses that basic type to generate therequired complex type object. Each transform function may include twoparts—an “import” part and a “code” part.

The “imports” part of the transform function enables the addition of theset of imports that the underlying transform functions make use of.These are the imports which are required to compile the code. The“imports” part may include a “built_in” section which includes all ofthe built-in imports that the transform function uses. If the transformfunction makes use of only these built-in or default library imports inthe snippets, then the “built_in” section can be omitted. In case of thesnippets using any imports apart from those present in the built-in ordefault library, those need to be mentioned here. The “imports” part mayalso include a “third party” section which allows users to add importsthat are not built into Java by default. If the programmer wants toimport a class or a package present in an external jar, the programmercan use this “third party” section to specify them. The “third party”section may include internal sections, such as a “mvn_link” whichidentifies the maven repository link for the jar and an “import_string”which identifies the classes or packages from the jar that need to beimported.

The “code” part of the transform function contains all the transformfunction snippets. In selected embodiments, each transform function canbe put under a “function_code” section which defines each function. Inthis example the function's modifiers should be public static. Inaddition, the provided transform function should be compilable when putinside a proper class with imports mentioned above.

To provide an improved understanding of selected embodiments of atransform function, reference is now made to an example “wildCardMatch”library function of FilenameUtils which has the following signature:

-   -   FilenameUtils.wildcardMatch(java.lang.String, java.lang.String,        org.apache.commons.io.IOCase).

In this example library function, the “org.apache.commons.io.IOCase”class is a complex object. An example usage of how custom inputs couldbe passed for that class is given below:

Custom_inputs:        - input:          - “abc.txt” # The first Stringparameter          - “*.txt” # The second String parameter          -type:[“java.lang.String”]# The third parameter is IOCase which is acomplex type. So we make use of the transform (contd.)          input_value:[“Sensitive”]# function ‘getIOCase’ which takes ina String (mentioned in the ‘type’ section) and the (contd.)          transform_function: getIOCase # String value it takes in is“Sensitive” (mentioned in the ‘input_value’ section)        - input:         - “one”          - “*.jpg”          - type:[“java.lang.String”]            input_value: [”Insensitive”]           transform_function: getIOCase transform_function_snippets:         imports:            built_in:       # optional               -java.nio.*               - java.io.*            third_party:              - mvn_link:http://mvnrepository.com/artifact/commons-io/commons-io/2.4  # mavenrepo link of apache commons-io jar                import _string:‘org.apache.commons.io.IOCase’ # The class that we want to import fromthe jar          code:           - function_code: | # The transformfunction that takes in a String and returns an IOCase Object              public static IOCase getIOCase(String name) throwsIOException {                IOCase ioCase = IOCase.forName(name);               return ioCase;               }

As seen from the foregoing example, a custom input can be specified byusing a transform function to define the “type,” the “input value” andthe “transform function.” The “type” defines the type of input that thetransform function receives as an input. The types here should bementioned as a fully qualified name similar to the signature part.Generally, these would be the simple Java types, like String, int,float, boolean, and their respective arrays. In order to specify acustom input, the transform function should also specify the actual“input value” that is passed to the parameter of the transform functionin order. In selected embodiments, the “input value” can have decoratorswhich will transform the data applied to it. For example, a “FILE”decorator that is applied over any data (e.g., “file_data”) would storethe data contents in a file and return a file name (e.g., “file_name”).As disclosed herein, the transform functions can then be written to takein that file name as a parameter and then try to generate complex types,assuming it has access to the file name of the file it needs to useinside its code. In selected embodiments, the data that needs to bepassed to the “FILE” decorator may be defined in a separate section(e.g., file_data) whereby data passed to files can be accessed like anarray to get its contents. For example, a use case scenario of using atransform function to specify custom inputs would be an input value,input_value: [FILE(file data[0])]. In this case, the final value for the“input_value” would be a string which is nothing but a file name, andthat file's contents would be the data present in the zeroth index ofthe file_data section. This file name can then be passed to a transformfunction as a String and that transform function can in turn have codethat makes use of that file Finally, the “transform function” is thename of the transform function to call to get the actual input.

Continuing with this example, in order to pass in InputStream as aninput to the function IOUtils.toString(java.io.InputStream), thefollowing transform function could be used:

signature:       method_parameter_types:         - java.io.InputStream      return_type:         - java.lang.String custom_inputs:       -input:         - type: [“java.lang.String”]           input_value:[”FILE(file_data[0])”]           transform_function: getInputStreamfile_data:       - Sample data for the file transform_function_snippets:        code:           - function_code: |             public staticInputStream getInputStream(String filename) throws IOException {              File file = new File(filename);               InputStreamis = new FileInputStream(file);               return is;             }

In the foregoing example, the transform function is written in to expecta filename as a String, and to create a new File object from which itcreates an InputStream object. The FILE decorator may be used toimplement this by using the FILE decorator (which returns a file name inthe back end) to pass the file name returned from it to a transformfunction which can, in turn, use it however it wants. In addition, theFILE decorator has the ability to specify the contents of the file.

As reference above, the “File Data” section can be used to define datathat could be used as contents of a file. These file datum can then beused while providing custom inputs as part of the FILE decorator. Theycan be accessed like an array. For example, file_data[0] would point tothe first file data entry, file_data[1] would point to the second filedata entry, and so on. The “File Data” section may be optional, butshould be used whenever FILE decorator is used in order to specify thefile data.

In use case of scenarios where the input type is some form of stream orreader (e.g., InputStream or BufferedReader) which is mostly generatedfrom the contents of a file, the end user can design the transformfunctions in a way that the end user just passes the contents of thefile and wraps it with the decorator FILE (which means that the engineshould generate a filename with the contents of the file being thecontents enclosed in the FILE decorator). The transform function in turncan be modelled to take in a fileName String and then write his ownsnippet to transform that to an InputStream or Reader, etc.

As disclosed herein, a good custom input set will have a number ofproperties. First, the custom inputs should try to cover all kind ofscenarios. Second, the custom inputs should cover edge cases. Third, incase of a boolean output, the number of inputs yielding True resultsshould be equal to the number of inputs yielding False results. Fourth,all custom inputs should be unique. Fifth, there should be more thanfive custom inputs.

“Equals True” and “Equals False” Conditions

As disclosed herein, the disclosed configuration file structure may alsoinclude “equals true” and “equals false” conditions data structures withfields identifying one or more conditions that, when evaluating to Trueand False, respectively, will be used to match the output generated bycode snippet and the expected output. These specified conditions arespecific to data type which being matched. In an example embodiment, an“AND” condition is specified with commas in the list, while an “OR”condition is specifically mentioned in the list. In addition, thedefault condition is considered as false. These conditions are Pythonboolean commands.

Code Snippets

The disclosed configuration file structure may also include code snippetdata structures which store Java code snippets which are used by thefuzzy/internal matching engine to find patterns when matching sourcecode with candidate library function substitutions. Each stored codesnippet should be compilable when put inside a proper class. Inaccordance with selected embodiments disclosed herein, each code snippetdata structure may have a mandatory “code” part and an optional“imports” part. The “code” section contains the function snippets whichperform similar tasks as that of the library. In selected embodiments,the “code” section has a “function_code” section where each function canbe defined. In this example, the function's modifiers should be publicstatic. The “imports” section identifies the imports which are requiredto compile the code. In selected embodiments, a list of default Javalibrary imports may be supported and stored for retrieval and access.For example, if the proposed library function uses any of the defaultimports (e.g., Java IO Library, Java Util Library, Java Math Library, orJava Net Library), then there is no need to specify them in the importssection of the code_snippet.

Since the logic of a proposed library function can be represented as acode snippet in an infinite number of ways, some rules should befollowed to prepare an optimal number of well-written code snippets andvariations thereof needed to submit for use by the library suggestionengine. When evaluating the quality of code snippets, a structuralsimilarity concept should be considered which compares the abstractsyntax tree similarity of code snippets. In addition, a logic similarityor variation concept should be considered which compares the codesnippets to determine if they are different from each other in terms oflogic and implementation.

As disclosed herein, a good set of code snippets will have a number ofproperties. First, each code snippet should be compilable (with properclass definition and imports). Second, any Variable and Method Namesshould be relevant to the context so as to refrain from using variables(e.g., “i”, “foo”, “bar”, etc.) and instead use the camelcase javaconvention (e.g., “filename”, “fileContents”, etc.). In addition, thewriter should use variable names that a developer would while using thelibrary functions. Third, each code snippet should do only what theproposed library function does, so that there is no additional pre- orpost-processing. To achieve this, the writer should avoid null checks,print statements, and any other such statements that do not have anyimpact on the core logic of the function. Fourth, each code snippetshould be structurally different from the other code snippets. Inmeeting this requirement, the variables should not be renamed whencreating a new code snippet, and re-ordering of statements whereverpossible should be included. Fifth, there should be, for each variationof logic, at least two structurally different code snippets provided.Sixth, all possible logic variations should be incorporated in the codesnippets.

Sample Configuration File

In accordance with the foregoing use case scenario for a libraryconfiguration file for the following method signature:

public static String toString(Reader input)             throwsIOException,the sample configuration file should be:

--- library_name: “org.apache.commons.io” # The package namelibrary_function_name: “org.apache.commons.io.IOUtils.toString” # Thefunction name including class name as well library_code_link:“https://github.com/apache/commons-io” documentation_link:“https://commons.apache.org/proper/commons-io/apidocs/org/apache/commons/io/IOUtils.html#toString(java.io.Reader)”signature:       method_parameter_types:         - java.io.Reader      return_type:         - java.lang.String custom_inputs:        -input:          - type: [“java.lang.String”]            input_value:[“Good Morning”]            transform_function: getReadertransform_function_snippets:         imports:           built_in:            - java.io.*         code:           - function_code: |             public static Reader getReader(String              data)  {               StringReader reader = new StringReader(data);               return reader;              } equal_function_True: [output == expected_output or output.strip( ) == expected_output.strip( )or ‘‘.join(output.splitlines( )) == ’’.join(expected_output.splitlines())] equal_function_False: [ output != expected_output, output is None orexpected_output is None ] code_snippets:          imports:            built_in:                - java.io.*                -java.util.*          code:            - function_code: |            public static String lib_readFile(InputStreamReader is)throws IOException {               StringBuilder s = new StringBuilder();               BufferedReader br = new              BufferedReader(is);               while (br.ready( )) {                s.append(br.readLine( ));                s.append(“\n”);               }               String s2= s.toString( );               br.close( );               return s2;            }           - function_code: |             public staticString lib_readFile(Socket sock, String cmd) throws IOException {              OutputStream outstream = sock.getOutputStream( );              outstream.write(cmd.getBytes( ));              outstream.flush( );               sock.shutdownOutput( );              BufferedReader reader = new               BufferedReader(                    new InputStreamReader(sock.getInputStream( )));              StringBuilder sb = new StringBuilder( );              String line;               while((line = reader.readLine()) !=               null) {                 sb.append(line + “\n”);              }               return sb.toString( );             }          - function_code: |             public static Stringlib_readFile( ) throws IOException {               String s = “”;              BufferedReader br = new BufferedReader(new FileReader(newFile(“ngramsy.txt”)));               while (br.ready( )) {                s += br.readLine( );                 s += “\n”;              }               br.close( );               return s;              } ...

Automation of Library Model Generation

To address the technical challenges that users experience when uploadinga library file in the correct YAML file format and thereby facilitatethe library model addition process, the assembly and submission of thelibrary configuration file 232 during the library configuration model231 input step in the workflow 200 may include an automated process forextracting or scraping information to automatically generate the libraryconfiguration file 232 in the correct format.

As a preliminary step in automating the generation of the libraryconfiguration file, a basic template is defined for a YAML libraryconfiguration file 232. In selected embodiments, any suitable templatingengine (such as the Jinja or Jinja2 python template engine) may be usedto define a basic YAML file template as follows:

--- library_name: “{{library_name}}” library_function_name:“{{library_name}}.{{function_name}}” documentation_link:“{{documentation_link}}” library_code_link: “{{library_code_link}}”signature:       method_parameter_types:         {% for i infunction_parameters -%}         - {{i}}       {% endfor %}      return_type:         - {{return_type}} equal_function_True: [output == expected_output] equal_function_False: [ output !=expected_output, output is None or expected_output is None ]code_snippets: {code: {{code}}} description:   code:“{{function_name}}({{function_parameters_string}})”   advantages:    -name: “Maintainability”      rating: 3      description: “Code is lesscomplex, clearer and easy to understand since Apache Commons Lang is apopular library.Hence the code becomes more maintainable”    - name:“Robustness”      rating: 4      description: “They have a very activecommunity and hence the chances of bugs in the library code is less andthe fixes are frequent making the code robust”    - name:“Documentation”      rating: 3      description: “There is complete andample documentation given for the given library”    - name: “TestCoverage”      rating: 3      description: “This library has acomprehensive set of developer tests, providing assurance of the libraryquality”    - name: “Maturity”      rating: 3      description: “Thelibrary has been under active development since 2002. It is a mature andreliable code base” code_type: “{{code_type}}” maven_link:“{{maven_link}}” logo_image_link: “{{logo_image_link}}” summary:“{{description}}” number_of_lines: 1 ...

Continuing with this example YAML file template, the libraryconfiguration model 231 may include an extraction module (e.g.,extractor 17 in FIG. 1) which is configured to extract the“library_name” and “function_name” parameters from the documentationlink identified at the “document_link” parameter. In addition, theextraction module may be configured to extract the “function_parameters”from the content of the page. For example, the extraction module scrapesthe web page to find where the tag corresponding to the particularfunction is present, and then uses the tag to extract the return typeand parameters which are passed to it. The extraction module works in asimilar fashion to extract the “return_type” parameter from the page. Inorder to assemble the “transform_function” parameters, the extractionmodule may be configured to collect basic transform function and addthem as a source project to an online hosting service (such as GitHub),and then use the “function_parameter” and “return_type” parameters tosearch the hosting service and retrieve it. The “description” parametermay be assembled by using the extraction module to scrape the web pageof the given library function. In order to extract the “code_type”parameter, the “library class_name” parameter may be mapped to the“code_type” parameter which can then be extended by the extractionmodule. And for the “code” parameters which provide the code snippetsused for fuzzy/internal matching, the extraction module may beconfigured to extract these snippets by scraping repositories at anonline hosting service (e.g., GitHub). The extraction module may alsoextract the “custom_input_data” parameters from test cases of the givenlibrary function, thereby providing custom input test cases. If alibrary logo image link is to be included in the library configurationfile, the extraction module may be configured to extract a“logo_image_link” parameter from an internally stored dictionary. Insimilar fashion, the extraction module may be configured to access aninternally stored dictionary to retrieve a “library_code_link” parameterthat provides a code link to the library name mapping and/or a“maven_link” parameter specifying a Mavenlink repository location forthe library configuration file.

As disclosed herein, the extraction module may conduct an iterativesequence of operations to retrieve, scrape or extract informationparameters to populate data fields in the YAML library configurationfile 232. In an example of a sample model for the library function“StringUtils.swapCase” present in the apache commons library which swapsthe case of the characters in the string (e.g., converts the lowercaseletters of the String to uppercase and vice versa), a first iterativepass may generate an initial version of the YAML library configurationfile as set forth below:

--- library_name: “org.apache.commons.lang3” library_function_name:“org.apache.commons.lang3.StringUtils.- swapCase” documentation_link:“https://commons.apache.org/proper/commons-lang/apidocs/org/apache/commons/lang3/StringUtils.html#swapCase-java.lang.String-” library_code_link:“https://github.com/apache/commons-lang” signature:      method_parameter_types:         - java.lang.String      return_type:         - java.lang.String custom_inputs:         -input:           - “abc”         - input:           - “aBc”         -input:           - “The dog has BONE”         - input:           - “CAT”        - input:           - “Cat” equal_function_True: [ output ==expected_output ] equal_function_False: [ output != expected_output,output is None or expected_output is None ] code_snippets: {code: [ ]}description:   code: “StringUtils.swapCase(java.lang.String)”  advantages:    - name: “Maintainability”      rating: 3     description: “Code is less complex, clearer and easy to understandsince Apache Commons Lang is a popular library.Hence the code becomesmore maintainable”    - name: “Robustness”      rating: 4     description: “Apache Foundation is a very active community andhence the chances of bugs in the library code is less and the fixes arefrequent making the code robust”    - name: “Documentation”      rating:3      description: “There is complete and ample documentation given forApache Commons Lang Library”    - name: “Test Coverage”      rating: 3     description: “Apache Commons Lang has a comprehensive set ofdeveloper tests, providing assurance of the library quality”    - name:“Maturity”      rating: 3      description: “The library has been underactive development since 2002. It is a mature and reliable code base”code_type: “String Manipulation” maven_link:“http://mvnrepository.com/artifact/org.apache.commons/commons-lang3/3.4” logo_image_link:“http://s20.postimg.org/4g0cz3ayh/apache.png” summary: “Swaps the caseof a String changing upper and title case to lower case, and lower caseto upper case.” number_of_lines: 1 ...

As seen above, the initial version of the YAML library configurationfile does not include data for the “code” entry for the “code_snippets”data field. To retrieve these code snippets, the extraction module maybe configured to conduct a second iterative pass to find the codesnippets by scraping an online repository hosting service (e.g., GitHub)using custom inputs extracted for input/output matching from the onlinerepository. In addition or in the alternative, the “custom_inputs”present in the initial version of the YAML library configuration filecontains some sample test cases for the library function. By passingthese test cases to the input/output matching engine (e.g., blackboxmatching engine 228) in a second iterative pass, they may be analyzed ona repository of open source projects by injecting these test cases intothe codes of each of the projects. The result can then be used toautomatically identify snippets that could be used as sample codesnippets for the library model. As a result of the second iterativepass, the final YAML library configuration file may be generated as setforth below:

--- library_name: “org.apache.commons.lang3” library_function_name:“org.apache.commons.lang3.StringUtils.swapCase” documentation_link:“https://commons.apache.org/proper/commons-lang/apidocs/org/apache/commons/lang3/StringUtils.html#swapCase-java.lang.String-” library_code_link:“https://github.com/apache/commons-lang” signature:      method_parameter_types:         - java.lang.String      return_type:         - java.lang.String custom_inputs:         -input:           - “abc”         - input:           - “aBc”         -input:           - “The dog has BONE”         - input:           - “CAT”        - input:           - “Cat” equal_function_True: [ output ==expected_output ] equal_function_False: [ output != expected_output,output is None or expected_output is None ] code_snippets:         code:          - function_code: |             public static StringswapCase1(String string_val) {               if(string_val == null ||string_val.length( ) == 0) {                 return string_val;              }               final char[ ] buffer =string_val.toCharArray( );               for (int i = 0; i <buffer.length; i++) {                 final char ch = buffer[i];                if (Character.isUpperCase(ch)) {                  buffer[i] = Character.toLowerCase(ch);                } else if (Character.isTitleCase(ch)) {                  buffer[i] = Character.toLowerCase(ch);                } else if (Character.isLowerCase(ch)) {                  buffer[i] = Character.toUpperCase(ch);                }               }               return newString(buffer);             }           - function_code: |              public static String swapCase2(String string_val) {                if(string_val == null) {                   return null;                } else if(string_val.length( ) == 0) {                  return “”;                 }                 finalchar[ ] buffer = string_val.toCharArray( );                 for (int i =0; i < buffer.length; i++) {                   final char ch =buffer[i];                   if (Character.isUpperCase(ch)) {                    buffer[i] = Character.toLowerCase(ch);                  } else if (Character.isTitleCase(ch)) {                    buffer[i] = Character.toLowerCase(ch);                  } else if (Character.isLowerCase(ch)) {                    buffer[i] = Character.toUpperCase(ch);                    }                   }                   return newString(buffer);                 }             - function_code: |                public static String swapCase3(String string_val) {                  if(string_val != null && string_val.length( ) != 0) {                    final char[ ] buffer = string_val.toCharArray( );                    for (int i = 0; i < buffer.length; i++) {                      final char ch = buffer[i];                      if (Character.isUpperCase(ch)) {                        buffer[i] = Character.toLowerCase(ch);                      } else if (Character.isTitleCase(ch)) {                        buffer[i] = Character.toLowerCase(ch);                      } else if (Character.isLowerCase(ch)) {                        buffer[i] = Character.toUpperCase(ch);                      }                     }                     returnnew String(buffer);                   }                   returnstring_val;                 }             - function_code: |                public static String swapCase4(String string_val) {                      if(string_val != null && string_val.length( ) !=0) {                         final char[ ] buffer =string_val.toCharArray( );                         for (int i = 0; i <buffer.length; i++) {                           final char ch =buffer[i];                           if (Character.isUpperCase(ch)) {                            buffer[i] = Character.toLowerCase(ch);                          } else if (Character.isTitleCase(ch)) {                            buffer[i] = Character.toLowerCase(ch);                          } else if (Character.isLowerCase(ch)) {                            buffer[i] = Character.toUpperCase(ch);                          }                         }                        return new String(buffer);                      } else if(string_val == null) {                        return null;                       }                      return string_val;                     }                - function_code: |                     public staticString swapCase5(String string_val) {                      if(string_val != null && string_val.length( ) !=0) {                         final char[ ] buffer =string_val.toCharArray( );                         for (int i = 0; i <buffer.length; i++) {                           final char ch =buffer[i];                           if (Character.isUpperCase(ch)) {                            buffer[i] = Character.toLowerCase(ch);                          } else if (Character.isTitleCase(ch)) {                            buffer[i] = Character.toLowerCase(ch);                          } else if (Character.isLowerCase(ch)) {                            buffer[i] = Character.toUpperCase(ch);                          }                         }                        return new String(buffer);                      } else if(string_val != null) {                        return “”;                       }                      return null;                     } description:  code: “StringUtils.swapCase(java.lang.String)”   advantages:    -name: “Maintainability”      rating: 3      description: “Code is lesscomplex, clearer and easy to understand since Apache Commons Lang is apopular library.Hence the code becomes more maintainable”    - name:“Robustness”      rating: 4      description: “Apache Foundation is avery active community and hence the chances of bugs in the library codeis less and the fixes are frequent making the code robust”    - name:“Documentation”      rating: 3      description: “There is complete andample documentation given for Apache Commons Lang Library”    - name:“Test Coverage”      rating: 3      description: “Apache Commons Langhas a comprehensive set of developer tests, providing assurance of thelibrary quality”    - name: “Maturity”      rating: 3      description:“The library has been under active development since 2002. It is amature and reliable code base” code_type: “String Manipulation”maven_link:“http://mvnrepository.com/artifact/org.apache.commons/commons-lang3/3.4” logo_image_link:“http://s20.postimg.org/4g0cz3ayh/apache.png” summary: “Swaps the caseof a String changing upper and title case to lower case, and lower caseto upper case.” number_of_lines: 1

Validation of Library Configuration File

To address the technical challenges that users experience when uploadinga proposed library file in the correct YAML file format and preventknowledge base corruption from improperly uploaded files, the assemblyand submission of the library configuration file 232 during the libraryconfiguration model 231 input step in the workflow 200 may include anautomatic validation process to make sure that the library configurationfile 232 is in the correct format and satisfies predeterminedconstraints required for the library suggestion engine 201. For example,selected embodiments of the present disclosure employ a YAML validatorwhich uses a python package to validates whether the content of the YAMLlibrary configuration file 232 is correct or not. While validationprocessing to check a YAML file with normal strings is relativelystraightforward, the complexity of the processing task increases whenvalidating the code snippets 234 and the transform function snippets forthe custom inputs 235 in the library configuration model 231. However,by properly templatizing and compiling the library functions, thecorrectness of the code snippets can be checked. To this end, thelibrary model addition engine may include a YAML validator (e.g.,validator 18 in FIG. 1) which is stored locally or hosted on anaccessible server for installation and local use. In operation, the YAMLvalidator may be configured to fill-in predetermined data structurefields of the YAML library configuration file 232, such as by scrapingthe webpage of a given library or otherwise populating the YAML libraryconfiguration file with specified values. In addition, the YAMLvalidator may be configured to perform multiple different validationoperations on the submitted YAML library configuration file so as tocheck the YAML format, check the YAML file fields for correctness, checkthe validity of code snippets and transform function snippets, such asby removing false positives from the custom inputs for the input/outputmatching engine.

In accordance with selected embodiments of the present disclosure, theYAML validator may be configured to perform YAML format validation testto determine if the format of the YAML library configuration file 232 isfine or not. In addition, the YAML validator may be configured to checkwhether the fields provided in the YAML library configuration file 232are correct. In selected embodiments, the file validity check may beperformed with a suitable Python data validation library, such asVoluptuous (https://github.com/alecthomas/voluptuous). In addition, theYAML validator may be configured to confirm that the links in the YAMLlibrary configuration file 232 (e.g., the documentation link, mavenlink, etc.) are working links. In addition, the YAML validator may beconfigured to determine whether the number of code snippets is greaterthan a specified minimum or threshold needed for decent quality. Inselected embodiments of the YAML validator, the YAML libraryconfiguration file 232 is checked for URL validation, logo image linkvalidation, supported types validation, file data validation, transformfunction validation, custom inputs type validation, supported returntype validation, and/or library description basic validation. While mostof the type validation can be implemented with the Voluptuous datavalidation library, support has been added in Voluptuous fornon-supported type validations, with contributions for the same beingchecked in athttps://github.com/alecthomas/voluptuous/graphs/contributors.

In accordance with selected embodiments of the present disclosure, theYAML validator may be configured to perform validation testing of thecode snippets in the YAML library configuration file 232 that are usedfor fuzzy or internal match processing. Such validity testing mayinclude checking that, given suitable imports and proper class names,all the code snippets are individually compilable. If the code snippetsare not compilable, the YAML validator indicates that the YAML libraryconfiguration file is not correct.

In accordance with selected embodiments of the present disclosure, theYAML validator may be configured to perform validation testing of thetransform functions, custom inputs, and method signature in the YAMLlibrary configuration file 232 that are used for input/output matchprocessing. Such validity testing may include checking that if thetransform function snippets are compilable and/or not in accordance withthe method signature. In addition, the YAML validator may be configuredto check if the custom inputs provided are correct (i.e., satisfy themethod signature). If not, the YAML validator indicates that the YAMLlibrary configuration file is faulty. The YAML validator may also beconfigured to check that, given the JAR for a library function, outputscan be generated with respect to a given set of custom inputs.

In accordance with selected embodiments of the present disclosure, theYAML validator may be configured to perform validation testing of thecustom inputs and outputs in the YAML library configuration file 232that are used for input/output match processing by removing falsepositives from the custom input/outputs. In order to increase theaccuracy of the input/output match processing, the test cases should begenerated with respect to the parameters required by the libraryfunction. While one approach for tackling this problem is to generaterandom test cases with respect to a particular type (e.g., for a“String” type, the generated test cases could be “ ”, “a”, “ab”, “abc”,etc), there are drawbacks to this approach since it does not take intoconsideration the metadata involved in the library function. Forexample, an apache commons function, like isExtension, which takes intwo strings as inputs and checks if the second input string is extensionof first input string, the validation test should ensure that thecontext remains so that, for example isExtension(‘file.txt’, ‘txt’)makes more sense than isExtension(‘a’,‘b’). In order to obtain thecontext, the YAML validator may be configured to scrape the test casesprepared for a given library function by retrieving the source of entirelibrary and then heuristically checking if a given file is a test fileor not. If the file is a test file, then the validator checks for theusage of isExtension in the file and then scrapes the correspondinginputs. By doing this, the number and quality of custom inputs issignificantly increased, thereby increasing the accuracy of theinput/output match processing engine.

By using the YAML validator to check the YAML library configuration file232 for correct formatting, code snippets, transform functions, custominputs, and method signature validity, developers can smoothly generateand test the library models locally on their system.

To provide additional details for an improved understanding of selectedembodiments of the present disclosure, reference is now made to FIG. 3which depicts a simplified flow chart 300 showing the logic forsubmitting validated library models which may be recommended for libraryfunction substitutions to a developer. The processing shown in FIG. 3may be performed by a cognitive system, such as the computing system 100shown in FIG. 1 or other natural language processing system.

At step 301, a candidate library model is created, written, or retrievedby a developer. At this point in the software development process, thecandidate library model code has not been validated for addition to thelibrary knowledge base.

Using data extracted from or describing the candidate library model, alibrary configuration file is automatically created and loaded at step302. In selected embodiments, the library configuration file may beautomatically generated at step 302 by filling a configuration filetemplate with extracted data using an iterative sequence of operations302A-B to retrieve, scrape or extract data to automatically populate toa plurality of data fields in the configuration file template whichcontain information identifying the library model, a set of custominputs for the library model, and a set of code snippets that arefunctionally similar to the library model. The assembled libraryconfiguration file may include library function description information303, such as the library name, library function name, librarydescriptors, related documentation links, and method signature for thelibrary function. The library configuration file may also include custominput information 304 that is used for input/output matching. Inaddition, the library configuration file may include code snippetinformation 305 that is used for fuzzy or internal matching. Inaccordance with the present disclosure, the code snippets 305 in eachconfiguration file for use with evaluation by the internal matchingengine may be automatically identified by using library function testcases 305 to analyze open source projects and thereby identify thefunctionally similar code snippets. In selected embodiments, the libraryconfiguration file may be structured as a human-readable dataserialization language, such as YAML.

At step 306, the configuration file is submitted to the library modeladdition engine. At this point, the library configuration file for thecandidate library model may be evaluated to ensure that the file is inthe correct format and satisfies the constraints provided by the libraryrecommendation engine. In selected embodiments, a YAML validator may beimplemented as a python package which validates whether the content of aYAML library configuration file is correct or not, and to determine ifthe code snippets and transform function snippets from the libraryconfiguration file may be compiled.

Once validated, the approved candidate library model is uploaded to thelibrary knowledge base, such as the Amazon Simple Storage Service (S3)(step 308) where it may be used to generate customized code suggestionsfor library function substitutions for a programmer's submitted sourcecode. In particular and as described more fully below, the approvedcandidate library model includes library function information andlibrary function code snippets that may be used in the libraryrecommendation process steps 310, 320, 330, 340.

For example, the library recommendation process begins at step 310 whenthe source code files are written or retrieved by a developer, and thenchecked in or committed. At this point in the software developmentprocess, the source code includes untested code that may be fragile,bloated, untested, and low quality code which contains undiscovered bugsand is otherwise inefficient and not readily readable.

At step 320, input source code files are pruned or filtered to identifysource code files that are likely candidates for library functionsuggestions while removing source code files that do not have certainkeywords that are most likely to qualify for library functionsuggestions. The processing performed at step 320 may use machinelearning, natural language processing, and/or artificial intelligencetechniques to find keywords in the input source code files and thenassign a priority or weight value. In selected embodiments, the pruningstep 320 may be implemented with a feature vector generation step 321which uses vector formation techniques to generate feature vectors foreach input source code file. Using the generated feature vectors alongwith library function feature vectors retrieved from memory, candidatecode snippets from the input source code files may be identified at step322 by comparing the input source code file feature vectors and thelibrary function feature vectors to generate a numerical similaritymeasure that may be compared with a pruning threshold values for thecorresponding library function. The computation steps performed at step322 to identify similarities between the feature vectors may includetokenizing input code snippets and code snippets from the libraryfunctions to generate comparative file vectors which are evaluated(e.g., by dot product) against a pruning threshold to identify candidatecode snippets, checking for the presence of predetermined words in theinput code and assigning a corresponding weight, or by any othersuitable code filtering operations for identifying candidate codesnippets from the input code that should be further processed forlibrary suggestion opportunities.

At step 330, the pruned input source code files (e.g., candidate codesnippets) are validated and matched with library function informationcontained in the knowledge base to identify validated source code files(e.g., validated code snippets). The processing performed at step 320may use machine learning, natural language processing, and/or artificialintelligence techniques in combination with static and/or dynamic codeanalysis to identify and validate input source code files that likelyqualify for library function suggestions. In selected embodiments, thevalidation and matching step 330 may be implemented with a first matchprocessing step 331 which matches code snippet features extracted froman input source code file with extracted library function features, suchas by using abstract syntax tree and/or Ngram execution flow matchingalgorithms. In effect, the first match processing step 331 performsfuzzy matching of the structural syntax and/or bytecode execution flowusing automated matching threshold values (e.g., AST size and distancethresholds and/or Ngram thresholds).

In selected embodiments, the validation and matching step 330 may alsoinclude a second match processing step 332 which performs input/outputmatching by injecting shared inputs to input code snippets and libraryfunction code snippets to detect matching outputs, thereby generatingvalidated code snippets which can be replaced by a library function. Ineffect, the second match processing step 332 performs exact matching toconfirm that the code snippets are the same if the same input yields thesame outputs. In selected embodiments, the second match processing step332 may be configured to extract library signature information from alibrary function, and to then extract candidate code snippets. Theextracted candidate code snippets may then be templated according to theextracted library signature and then compiled so that a shared input isinjected into the compiled code to generate outputs which are comparedto outputs generated from the library function on the basis of theshared input.

At step 340, the library function substitutions are recommended for eachvalidated source code file (e.g., validated code snippets) so that theprogram developer is presented with one or more library functionrecommendations which include may include code lines from input sourcecode files along with code lines from the library function suggestion,alone or in combination with additional library function informationidentifying the code improvement recommendation and/or code reductionresulting from the library function recommendation and/or educationaltutorial information relating to the implementation of the libraryfunction recommendation. The processing performed at step 340 may beperformed by a cognitive system, such as the computing system 100 shownin FIG. 1 or other data processing system functionality for displayinguser interface information. In selected embodiments, the recommendationstep 340 may be implemented with a first processing step 341 whichidentifies a code improvement and/or code reduction from the librarysubstitution recommendation, such as by quantifying a performancebenefit or potential code reduction that would result from using thelibrary function recommendations. In addition, the recommendation step340 may include a second processing step 342 which presents the librarysubstitution recommendation to the developer, alone or in combinationwith information about a code improvement or code reduction. As will beappreciated, the library substitution recommendation presented at step342 may be included in the same user interface screen or a differentuser interface screen in which the code improvements and/or codereductions are identified for the developer. In addition, therecommendation step 340 may include a third processing step 343 whichpresents the developer with an education tutorial about the librarysubstitution recommendation, such as by displaying the recommendedreplacement code for recommended library function along an explanationof the benefits of the recommended library function, a link to thesuggested library function, and a video explaining how to implement thesuggested library function.

To provide additional details for an improved understanding of selectedembodiments of the present disclosure, reference is now made to FIGS.4A-C which depict an example sequence of user interface screen shots fora library suggestion engine in accordance selected embodiments of thepresent disclosure. In the first example screen shot of user interface400 shown in FIG. 4A, there is displayed is a shown a summary oroverview for the evaluation of a plurality of input source code filesfor library suggestion recommendations. For example, the user interface400 shows a build status report 402, code quality report 403, unit testreport 405, and suggested library usage report 405 for a first inputsource code file (Build #113, Rev 4141) that has been committed orloaded into the library suggestion engine. As indicated in the libraryusage report 405, the user interface 400 indicates that one issue hasbeen detected and that the recommended library function will reduce thesize of the source code by four lines if selected by the developer forsubstitution or replacement.

By using the cursor 401 or other user interface controls to interactwith the user interface 400, the developer may cause the librarysuggestion engine to display a second user interface screen shot 410which illustrates a code reduction opportunity for the selected inputsource code file, as illustrated in FIG. 4B. In this example, the userinterface 410 shows a file identification field 412 for the first inputsource code file (e.g., Build #113, Rev 4141), an auto-classificationfield 413, code line replacement field 414, a code reduction field 415,and a library field 416. The file identification field 412 identifiesthe input source code file. The auto-classification field 413automatically shows that the first input source code file is a “fileoperation” file. The code line replacement field 414 shows the number ofcurrent code lines (in the validated code snippet) and the proposednumber of code lines (from using the recommended library function). Thecode reduction field 415 shows a quantification of the code reductionbenefit. And the library field 416 provides a link to additionalinformation for educating the developer about how to implement therecommended library function. Below the fields 412-416, the second userinterface screen shot 410 may also display the first input source codefile 417 with the validated code snippet 418 highlighted or otherwisevisually set off from the remaining lines of code in the input sourcecode file 417. In order to access additional information relating to therecommended library function, the user interface 410 may include one ormore user interaction links 419 in addition to the library field 416 sothat, when actuated by the cursor 401, additional information about therecommended library function may be displayed.

Referring now to FIG. 4C, there is shown a third user interface screenshot 420 which illustrates additional library information 421-424relating to the recommended library function for replacing the validatedcode snippet 418. In this example, the user interface 420 shows a firstfield 421 for identifying library code (e.g., IOUtils.toString of theApache Commons Library) that can be used to replace the validated codesnippet 418. The user interface 420 may also include a second field 422for describing the benefits of using the library functionrecommendation. The user interface 420 may also include a video tutorial423 and a link 424 to the library repository (e.g., maven repository).The video tutorial 423 can provide educational information to theprogrammer on how to replace the validated code snippet with therecommended library function and other information needed to migrate tothe recommended library function.

To provide additional details for an improved understanding of selectedembodiments of the present disclosure, reference is now made to FIG. 5which depicts an example user interface screen shot 500 of a libraryadministrator console which illustrates the automatically populatedlibrary configuration file data fields in accordance selectedembodiments of the present disclosure. In the example screen shot ofuser interface 500, a plurality of fields 501-508 provide informationabout the candidate library function having data fields that that may beautomatically populated, including a first data field 501 that ispopulated with the library name (e.g.,“apache.commons.lang.StringUtils.join”), a second data field 502 that ispopulated with the library type (e.g., “List Manipulation”), a thirddata field 503 that is populated with a description of the library logic(e.g., “The code snippet tries to convert a string array to a stringwith a conjunction.”), a fourth data field 504 that is populated withthe number of lines added (e.g., “1”), an optional fifth data field 505that is populated with an educational video link address (e.g.,“https://www.youtube.com/embed/7_doqq9zK2k”), an optional sixth datafield 506 that is populated with a maven repository link for thelocation where the library function is stored (e.g.,“http://mvnrepository.com/artifact/org.apache.commons/commons-lang3/3.4”),a seventh data field 507 that is populated with the method signature (aportion of which is displayed), and an eighth data field 508 that ispopulated with a sample usage snippet (e.g.,“StringUtils.join(array.separator)”). In addition, the ninth data field509 is populated with the comparison function (a portion of which isshown) for comparing the outputs of two functions with the input/outputmatching engine. Finally, the tenth data field 510 is populated withcode snippets with similar functionality to the candidate libraryfunction for use with a whitebox matching engine by using libraryfunction sample test cases as inputs to a blackbox matching engine toanalyze open source projects and thereby identify the functionallysimilar code snippets. Though not shown, it will be appreciated thatadditional data fields may be included, such as pruning thresholdvalues, pruning vectors, AST distance/size threshold values, Ngramthreshold values, etc. Once the library administrator console datafields are filled in, the submit button 511 may be activated to uploadthe library configuration file to the library model addition engine forvalidation processing. If the submitted information is validated, thenthe library may be added to the library knowledge base. However, ifissues arise in the validation process, they may be identified forresolution or correction by the user.

Embodiments of the system and method for recommending librarysubstitutions can be implemented on a computer system, such as ageneral-purpose computer 600 illustrated in FIG. 6. As disclosed thecomputer 600 includes input user device(s) 616, such as a keyboardand/or mouse, which are coupled to a bi-directional system bus 608. Theinput user device(s) 616 are used for introducing user input to thecomputer system 600 and communicating that user input to processor 602.The computer system 600 may also include a video memory 604, main memory606, and mass storage 618, all coupled to bi-directional system bus 608along with input user device(s) 616 and processor 602. The mass storage618 may include both fixed and removable media, such as other availablemass storage technology. Bus 608 may contain, for example, 32 addresslines for addressing video memory 604 or main memory 606. The system bus608 may also include, for example, an n-bit data bus for transferringdata between and among the components, such as CPU 602, main memory 606,video memory 604, and mass storage 618, where “n” is, for example, 32 or64. Alternatively, multiplex data/address lines may be used instead ofseparate data and address lines.

The computer 600 may also include I/O device(s) 610 which provideconnections to peripheral devices, such as a printer, and may alsoprovide a direct connection to remote server computer systems via atelephone link or to the Internet via an ISP. I/O device(s) 610 may alsoinclude a network interface device to provide a direct connection toremote server computer systems via a direct network link to the Internetvia a POP (point of presence). Such connection may be made using, forexample, wireless techniques, including digital cellular telephoneconnection, Cellular Digital Packet Data (CDPD) connection, digitalsatellite data connection or the like. Examples of I/O devices includemodems, sound and video devices, and specialized communication devicessuch as the aforementioned network interface.

Computer programs and data are generally stored as instructions and datain mass storage 618 until loaded into main memory 606 for execution.Computer programs may also be in the form of electronic signalsmodulated in accordance with the computer program and data communicationtechnology when transferred via a network. The method and functionsrelating to system and method for adding library models may beimplemented in a computer program for a library recommendation engine605.

The processor 602, in one embodiment, is a microprocessor manufacturedby Motorola Inc. of Illinois, Intel Corporation of California, orAdvanced Micro Devices of California. However, any other suitable singleor multiple microprocessors or microcomputers may be utilized. Mainmemory 606 is comprised of dynamic random access memory (DRAM). Videomemory 604 is a dual-ported video random access memory. One port of thevideo memory 604 is coupled to video amplifier or driver 612. The videoamplifier 612 is used to drive the display 614. Video amplifier 612 iswell known in the art and may be implemented by any suitable means. Thiscircuitry converts pixel data stored in video memory 604 to a rastersignal suitable for use by display 614. Display 614 is a type of monitorsuitable for displaying graphic images.

By now, it will be appreciated that there is disclosed herein a system,method, apparatus, and computer program product for enhancing operablefunctionality of a software program by performing a method at a devicehaving an operating system and system library. As disclosed, the system,method, apparatus, and computer program is operative to add a librarymodel to a library knowledge base by first receiving a software programto be submitted as a library model by a developer. From the librarymodel, a library configuration file is generated in a human-readableformat which is parsable by software (e.g., a YAML format libraryconfiguration file) by filling a configuration file template (e.g., aYAML format configuration file template) with extracted data toautomatically populate a plurality of data fields in the libraryconfiguration file with information identifying the library model and aset of custom inputs for the library model, and passing a set of testcases from the set of custom inputs to an input/output matching engineto evaluate a repository of code snippets (e.g., an online hostingservice, such as GitHub, for storing a plurality of open sourceprojects) and identify a set of functionally similar code snippets forinclusion in one of the data fields in the library configuration file.In selected embodiments, the library configuration file is generated byconducting an iterative sequence of operations to retrieve, scrape orextract data to automatically populate the plurality of data fields inthe YAML format configuration file template. For example, a firstiterative pass may be performed to generate an initial version of theYAML library configuration file template in which only a subset of theplurality of data fields is filled with extracted data, and then asecond iterative pass may be performed to generate a final version ofthe YAML library configuration file template in which one of the datafields in the YAML library configuration file is filled with the set offunctionally similar code snippets. In selected embodiments, theinformation identifying the library model includes a library name, alibrary function name, one or more library descriptors, a documentationlink for the library model, and a method signature for the librarymodel. In addition, the information identifying the library model mayinclude a first list of conditions that will determine a valid matchwhen the when a code snippet output evaluates to true and a second listof conditions that will determine a valid match when the when a codesnippet output evaluates to false. In selected example embodiments, thedescriptors include a library function code type descriptor, an embeddedvideo tutorial link descriptor, a maven repository link descriptor forthe library model, a library source code link descriptor for the librarymodel, a library function summary descriptor, a code line numberdescriptor for adding the library model, and a code snippet advantagesdescriptor for the library model. Once generated, the libraryconfiguration file is then submitted to a validator which validates thelibrary configuration file to ensure that the library model is correctlyformatted and satisfies predetermined library model constraints. Inselected embodiments, a YAML format library configuration file isevaluated with a YAML validator to ensure that the library model iscorrectly formatted and satisfies predetermined library modelconstraints. For example, the YAML validator may use a python packagefor checking that the YAML format library configuration file iscorrectly formatted, for performing a validity check on the set of codesnippets, and for performing a validity check on the set of custominputs to ensure that the library model satisfies predetermined librarymodel constraints. Finally, the library model is added to the libraryknowledge base if the library configuration file is correctly formattedand satisfies predetermined library model constraints.

The present invention may be a system, a method, and/or a computerprogram product such that selected embodiments include software thatperforms certain tasks. The software discussed herein may includescript, batch, or other executable files. The software may be stored ona machine-readable or computer-readable storage medium, and is otherwiseavailable to direct the operation of the computer system as describedherein and claimed below. In one embodiment, the software uses a localor database memory to implement the data transformation and datastructures so as to automatically generate and add libraries to alibrary knowledge base for use in detecting library substitutionopportunities, thereby improving the quality and robustness of softwareand educating developers about library opportunities and implementationto generate more readable, reliable, smaller, and robust code with lesseffort. The local or database memory used for storing firmware orhardware modules in accordance with an embodiment of the invention mayalso include a semiconductor-based memory, which may be permanently,removably or remotely coupled to a microprocessor system. Other new andvarious types of computer-readable storage media may be used to storethe modules discussed herein. Additionally, those skilled in the artwill recognize that the separation of functionality into modules is forillustrative purposes. Alternative embodiments may merge thefunctionality of multiple software modules into a single module or mayimpose an alternate decomposition of functionality of modules. Forexample, a software module for calling sub-modules may be decomposed sothat each sub-module performs its function and passes control directlyto another sub-module.

In addition, selected aspects of the present invention may take the formof an entirely hardware embodiment, an entirely software embodiment(including firmware, resident software, micro-code, etc.), or anembodiment combining software and/or hardware aspects that may allgenerally be referred to herein as a “circuit,” “module” or “system.”Furthermore, aspects of the present invention may take the form ofcomputer program product embodied in a computer readable storage mediumor media having computer readable program instructions thereon forcausing a processor to carry out aspects of the present invention. Thusembodied, the disclosed system, a method, and/or a computer programproduct is operative to improve the design, functionality andperformance of software programs by adding libraries for use inautomatically detecting and recommending library function substitutionsfor replacing validated code snippets in the software program.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a dynamic or static random access memory(RAM), a read-only memory (ROM), an erasable programmable read-onlymemory (EPROM or Flash memory), a magnetic storage device, a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a Public SwitchedCircuit Network (PSTN), a packet-based network, a personal area network(PAN), a local area network (LAN), a wide area network (WAN), a wirelessnetwork, or any suitable combination thereof. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Java, Python, Visual Basic.net,Ruby, Smalltalk, C++ or the like, and conventional proceduralprogramming languages, such as the “C” programming language, HypertextPrecursor (PHP), or similar programming languages. The computer readableprogram instructions may execute entirely on the user's computer, partlyon the user's computer, as a stand-alone software package, partly on theuser's computer and partly on a remote computer or entirely on theremote computer or server or cluster of servers. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a sub-system, module, segment,or portion of instructions, which comprises one or more executableinstructions for implementing the specified logical function(s). In somealternative implementations, the functions noted in the block may occurout of the order noted in the figures. For example, two blocks shown insuccession may, in fact, be executed substantially concurrently, or theblocks may sometimes be executed in the reverse order, depending uponthe functionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

The computer system described above is for purposes of example only, andmay be implemented in any type of computer system or programming orprocessing environment, or in a computer program, alone or inconjunction with hardware. Various embodiments of the present may alsobe implemented in software stored on a computer-readable medium andexecuted as a computer program on a general purpose or special purposecomputer. For clarity, only those aspects of the system germane to theinvention are described, and product details well known in the art areomitted. For the same reason, the computer hardware is not described infurther detail. It should thus be understood that the invention is notlimited to any specific computer language, program, or computer. Thesystem and method for adding library models may be implemented in anytype of computer system or programming or processing environment. It iscontemplated that the system and method for adding library models mightbe run on a stand-alone computer system, such as the one describedabove. The system and method for adding library models might also be runfrom a server computer systems system that can be accessed by aplurality of client computer systems interconnected over an intranetnetwork. Finally, the system and method for adding library models may berun from a server computer system that is accessible to clients over theInternet.

Benefits, other advantages, and solutions to problems have beendescribed above with regard to specific embodiments. However, thebenefits, advantages, solutions to problems, and any element(s) that maycause any benefit, advantage, or solution to occur or become morepronounced are not to be construed as a critical, required, or essentialfeature or element of any or all the claims. As used herein, the terms“comprises,” “comprising,” or any other variation thereof, are intendedto cover a non-exclusive inclusion, such that a process, method,article, or apparatus that comprises a list of elements does not includeonly those elements but may include other elements not expressly listedor inherent to such process, method, article, or apparatus.

1. A method for adding a library model to a library knowledge base,comprising: receiving, by a device comprising a processor and a memory,a software program to be submitted as a library model; generating, bythe device, a library configuration file by filling a configuration filetemplate with extracted data to automatically populate a plurality ofdata fields in the library configuration file with informationidentifying the library model and a set of custom inputs for the librarymodel, and passing a set of test cases from the set of custom inputs toan input/output matching engine to evaluate a repository of codesnippets and identify a set of functionally similar code snippets forinclusion in one of the data fields in the library configuration file;submitting, by the device, the library configuration file to a validatorwhich validates the library configuration file to ensure that thelibrary model is correctly formatted and satisfies predetermined librarymodel constraints; and adding the library model to the library knowledgebase if the library configuration file is correctly formatted andsatisfies predetermined library model constraints. 2.-20. (canceled)